An Approximate Shapley-Folkman Theorem

The Shapley-Folkman theorem shows that Minkowski averages of uniformly bounded sets tend to be convex when the number of terms in the sum becomes much larger than the ambient dimension. In optimization, Aubin and Ekeland [1976] show that this produces an a priori bound on the duality gap of separable nonconvex optimization problems involving finite sums. This bound is highly conservative and depends on unstable quantities, and we relax it in several directions to show that non convexity can have a much milder impact on finite sum minimization problems such as empirical risk minimization and multi-task classification. As a byproduct, we show a new version of Maurey's classical approximate Carath\'eodory lemma where we sample a significant fraction of the coefficients, without replacement, as well as a result on sampling constraints using an approximate Helly theorem, both of independent interest.Comment: Added constraint sampling result, simplified sampling results, reformat, et

Scaling-up Empirical Risk Minimization: Optimization of Incomplete U-statistics

In a wide range of statistical learning problems such as ranking, clustering or metric learning among others, the risk is accurately estimated by $U$-statistics of degree $d\geq 1$, i.e. functionals of the training data with low variance that take the form of averages over $k$-tuples. From a computational perspective, the calculation of such statistics is highly expensive even for a moderate sample size $n$, as it requires averaging $O(n^d)$ terms. This makes learning procedures relying on the optimization of such data functionals hardly feasible in practice. It is the major goal of this paper to show that, strikingly, such empirical risks can be replaced by drastically computationally simpler Monte-Carlo estimates based on $O(n)$ terms only, usually referred to as incomplete $U$-statistics, without damaging the $O_{\mathbb{P}}(1/\sqrt{n})$ learning rate of Empirical Risk Minimization (ERM) procedures. For this purpose, we establish uniform deviation results describing the error made when approximating a $U$-process by its incomplete version under appropriate complexity assumptions. Extensions to model selection, fast rate situations and various sampling techniques are also considered, as well as an application to stochastic gradient descent for ERM. Finally, numerical examples are displayed in order to provide strong empirical evidence that the approach we promote largely surpasses more naive subsampling techniques.Comment: To appear in Journal of Machine Learning Research. 34 pages. v2: minor correction to Theorem 4 and its proof, added 1 reference. v3: typo corrected in Proposition 3. v4: improved presentation, added experiments on model selection for clustering, fixed minor typo

Extending Gossip Algorithms to Distributed Estimation of U-Statistics

Efficient and robust algorithms for decentralized estimation in networks are essential to many distributed systems. Whereas distributed estimation of sample mean statistics has been the subject of a good deal of attention, computation of $U$-statistics, relying on more expensive averaging over pairs of observations, is a less investigated area. Yet, such data functionals are essential to describe global properties of a statistical population, with important examples including Area Under the Curve, empirical variance, Gini mean difference and within-cluster point scatter. This paper proposes new synchronous and asynchronous randomized gossip algorithms which simultaneously propagate data across the network and maintain local estimates of the $U$-statistic of interest. We establish convergence rate bounds of $O(1/t)$ and $O(\log t / t)$ for the synchronous and asynchronous cases respectively, where $t$ is the number of iterations, with explicit data and network dependent terms. Beyond favorable comparisons in terms of rate analysis, numerical experiments provide empirical evidence the proposed algorithms surpasses the previously introduced approach.Comment: to be presented at NIPS 201

Gossip Dual Averaging for Decentralized Optimization of Pairwise Functions

In decentralized networks (of sensors, connected objects, etc.), there is an important need for efficient algorithms to optimize a global cost function, for instance to learn a global model from the local data collected by each computing unit. In this paper, we address the problem of decentralized minimization of pairwise functions of the data points, where these points are distributed over the nodes of a graph defining the communication topology of the network. This general problem finds applications in ranking, distance metric learning and graph inference, among others. We propose new gossip algorithms based on dual averaging which aims at solving such problems both in synchronous and asynchronous settings. The proposed framework is flexible enough to deal with constrained and regularized variants of the optimization problem. Our theoretical analysis reveals that the proposed algorithms preserve the convergence rate of centralized dual averaging up to an additive bias term. We present numerical simulations on Area Under the ROC Curve (AUC) maximization and metric learning problems which illustrate the practical interest of our approach

Clustered Multi-Agent Linear Bandits

We address in this paper a particular instance of the multi-agent linear stochastic bandit problem, called clustered multi-agent linear bandits. In this setting, we propose a novel algorithm leveraging an efficient collaboration between the agents in order to accelerate the overall optimization problem. In this contribution, a network controller is responsible for estimating the underlying cluster structure of the network and optimizing the experiences sharing among agents within the same groups. We provide a theoretical analysis for both the regret minimization problem and the clustering quality. Through empirical evaluation against state-of-the-art algorithms on both synthetic and real data, we demonstrate the effectiveness of our approach: our algorithm significantly improves regret minimization while managing to recover the true underlying cluster partitioning.Comment: 18 pages, 8 figure

Communities of Practice in Landscapes of Practice

The original formulation of communities of practice primarily focused on describing how learning, meaning, and identity within a community can translate into a sustained practice. Wenger-Trayner et al. elaborated the concept of landscapes of practice to describe how different communities of practice may interact, and belong to broader landscapes of practice, rather than rely exclusively on their own local situated practices. In this conceptual article, we apply the perspective of landscapes of practice to organizations. The first part of our argument is descriptive, and is aimed at developing a model of landscapes of practice in organizations. With regard to this model, we propose that practices can be seen as multilevel, including local situated practices, generic practices, and cultural fields. This, in turn, helps to clarify and organize a number of central concepts within the practice literature. The second part of our argument is prescriptive, as we suggest that landscapes of practice call for triple-legitimization of situated learning, meaning that legitimization is not only needed at the level of community and organization, but also by attending to the dynamically changing epistemic texture of the landscapes

Knowledge Acquisition Using Group Support Systems

This paper reports on a project in which a group support system (GSS) equipped with a causal mapping facility was used to acquire knowledge from experts in seven European cities in order to understand the systemicity of risks which cities may face. The practical constraints demanded that participants’ experience and wisdom about the city risk environment was collected in a short period of time: three 1-day workshops. The acquisition of knowledge posed a number of important epistemological challenges which are explored in our discussion. The GSS was faced with the need to (1) facilitate sharing of knowledge with others, (2) manage the complexity of expert knowledge, (3) acknowledge the time demands on experts, (4) manage and merge multiple perspectives, and (5) acknowledge the subjectivity of knowledge in this domain. By discussing how the GSS process attended directly to these epistemological issues and to methodological considerations that linked to these issues, the paper contributes to a better understanding of the application of GSS for knowledge acquisition, particularly in comparison with other possible methods

Exploring GSS negotiation – the use of a GSS log

Group Decision Support Systems (GSS) have been used extensively to support groups in working together in organizations. This paper focuses on the particular type of GSS, called Group Explorer, which during the course of facilitated sessions generates data logs in the form of Excel spreadsheets. Data logs can be of high interest to researchers and GSS facilitators because they may possibly contain rich and valuable data such as about the detailed time of entry and the authorship of all contributions, or the results of voting activities conducted by participants. However, data logs may at first look complicated and difficult to read and follow. Thus the purpose of this paper is to provide a number of instructions and explanations for anyone interested in making good use of data logs, and to popularize similar analysis as a good opportunity to bet-ter understand the outcomes of GSS sessions